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Abstract 

Little is known theoretically about the associative memory capabilities of 
neural networks in which information is encoded not only in the mean firing 
rate but also in the timing of firings. Particularly, in the case that the fraction 
of active neurons involved in memorizing patterns becomes small, it is bio- 
logically important to consider the timings of firings and to study how such 
consideration influences storage capacities and quality of recalled patterns. 
For this purpose, we propose a simple extended model of oscillator neural 
networks to allow for expression of non-firing state. Analyzing both equilib- 
rium states and dynamical properties in recalling processes, we find that the 
system possesses good associative memory. 

84.35.+i,07.05.Mh,87.10.+e 



1 



Typeset using REVTeX 



Since several recent experiments suggest that the temporal coherence of neuronal activ- 
ity, such as the synchronization of pulses, plays a significant role in real neuronal systems 
JjJ, a great number of authors have proposed and studied many theoretical models of neu- 
ral networks in attempt to understand the essential dynamics of the these systems @-f||. 
Among these models, the neural networks of phase oscillators provide a useful framework 
for modeling and analyzing such temporal behavior in neuronal systems. The main reason 
for the usefulness of these models is their mathematical tractability, which has allowed us to 
obtain many important analytic results. Another reason is that, using a certain mathemati- 
cal technique, the complex dynamics of coupled oscillatory neuronal systems under suitable 
conditions can be reduced to the dynamics of phase oscillator PQ]. Since the relation be- 
tween real systems and these models is theoretically clear, it is expected that their analysis 
will shed light on the role of oscillatory behavior in real neuronal systems. In particular, 
the properties of oscillator neural networks with regard to associative memory have been 
studied recently by many authors [f7Hn]|. 

With regard to the above mentioned framework, it should be noted that all neurons are 
assumed to exhibit periodic firing states at all times. However, such an assumption is not 
realistic from a biological viewpoint. This is because, in real systems, whether or not a 
neuron is firing usually depends on the situation, in particular the pattern it is presently 
recalling. In fact, it is well known that only a small fraction of neurons are active at a given 
time in the central nervous system. Situations in which the level of activity is very low are 
often termed sparse coding |12|]. In the case of standard binary neurons, it is found that 
the storage capacity for such sparse coding diverges as — 1/alna, where a is the fraction of 
active neurons |T3|-|l6f. This is the optimal asymptotic form O]. It is biologically plausible 
that this theoretical optimal bound is achieved even for a simple Hebbian learning rule. 

In light of the above considerations, in order to construct more realistic models, it is 
natural to consider the encoding of information in both the mean firing rate and the relative 
timing of neuronal spikes. However, little is theoretically known about the properties of 
associative memory for sparse coded patterns in such systems. For example, one of the 
interesting questions is whether the storage capacity in such systems, as in the standard 
binary model, diverges as the activity level a decreases. Another essential point is the 
quality of the recalled pattern, particularly concerning the timing of the spikes. To clarify 
these points, we first need to extend the phase oscillator model to allow for expression of 



the non-firing state fllSfl . In this paper, we propose and examine a simple extended model as 
a first step toward the theoretical study of neuronal systems in which the timing of spikes 
can carry information. 

Let us start with a brief review of the theoretical basis of the phase oscillator model. It 
is well known that in coupled oscillatory neuronal systems, under suitable conditions, the 
original dynamics can be reduced theoretically to a simpler phase dynamics. The state of 
the zth neuronal oscillatory system can be then characterized by a single phase variable fa 
representing the timing of the neuronal firings. The typical dynamics of oscillator neural 
networks are described by equations of the form |5|,|E)|,|iTf| . 

dfa N 

— j = Ui + Jij sinOj -fa + (3ij). (0.1) 
Here, and are parameters representing the effect of the interaction. For simplicity, 
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we assume that all natural frequencies Ui are equal to some fixed value ujq. We can then 
eliminate u by applying the transformation fa — > tpi+uj^t. Using the complex representation 
Wi = exp(i<pi) and Cy = exp(z/3y) in (pTT|) , it is easily found that all neurons relax toward 
their stable equilibrium states, in which the relation Wi = hi/\hi\ (hi = Y^jLiCijWj) is 
satisfied. Following this line of reasoning, as a synchronous update version of the oscillator 
neural network we can consider the alternative discrete form [PHTTH . 



hit) N 
W(t+1) = ^ (°-2) 

Now we will attempt to construct an extended model of the oscillator neural networks 
to retrieve sparsely coded phase patterns. In equation ( p.2| ), the complex quantity hi can be 
regarded as the local field produced by all other neurons. We should remark that the phase 
of this field, /ij, determines the timing of the ith neuron at the next time step, while the 
amplitude \hi\ has no effect on the retrieval dynamics ( P-2| ). It seems that the amplitude can 
be thought of as the strength of the local field with regard to emitting spikes. Pursuing this 
idea, as a natural extension of the original model we stipulate that the system does not fire 
and stays in the resting state if the amplitude is smaller than a certain value. Therefore, we 
consider a network of N oscillators whose dynamics are governed by 

hit) N 

W i (t + l) = f(\h i {t)\)^ M*) = E C « W i(<)- (0-3) 

In this paper, we assume that f(x) = Q(x — H), where the real variable if is a threshold 
parameter and Q(x) is the unit step function; Q(x) = 1 for x > and otherwise. Accord- 
ingly, the amplitude |W/| assumes a value of either 1 or 0, representing the state of the ith 
neuron as firing or non-firing. Consequently, the neuron can emit spikes when the amplitude 
of the local field hi(t) is greater than the threshold parameter H. 

Now, let us define a set of P patterns to be memorized as £f = Afexp(i0f) (// = 
1, 2, . . . , P), where 9f and A? represent the phase and the amplitude of the ith neuron in 
the /ith pattern, respectively. For simplicity, we assume that the #f are chosen at random 
from a uniform distribution between and 2tt. The amplitudes Af are chosen independently 
with the probability distribution 

P(A?) = aSW - 1) + (1 - a)5(A?), (0.4) 

where a is the mean activity level in the patterns. Note that, if H = and a = 1, this model 
reduces to ( p.2|) . 

For the synaptic efficacies, to realize the function of the associative memory, we adopt 
the generalized Hebbian rule in the form 

where ^ denotes the complex conjugate of £j\ The overlap M^{t) between the state of the 
system and the pattern \i at time t is given by 
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1 N ~ 

M,(t) = m M (*)e^W = _ £^(t), (0.6) 

In practice, the rotational symmetry forces us to measure the correlation of the system with 
the pattern /i in terms of the amplitude component m^{t) = iM^t)]. 

Let us consider the situation in which the network is recalling the pattern £*;that is, 
mi (t) = m(t) ~ 0(1) and m M (t) ~ 0(l/y/N)(n ^ 1). The local field hi{t) in Eq.(gJ) can 
then be separated as 

N 

hi{t) = J2 C H W i(t) = mte^il + Zi(t), (0.7) 
i=i 



i N P 



where Zt(t) is defined by 

2*(*) = -4?EEeref^(*)- (o-s) 

The first term in Eq.( p.7| ) acts to recall the pattern, while the second term can be regarded 
as the noise arising from the other learned patterns. The essential point in this analysis is 
the treatment of the second term as complex Gaussian noise characterized by 

< Zi (t) >= 0, < \zi{t)\ 2 >= 2a(t) 2 . (0.9) 

In addition, we also assume that ipi(t) remains a constant, that is, <pi(t) = ipo- By applying 
the method of statistical neurodynamics to this model under the above assumptions p0|-|23|, 
we can study the retrieval properties analytically. As a result of such analysis we have found 
that the retrieval process can be characterized by some macroscopic order parameters, such 
as m(t) and cr(t). However, in this short paper, we give only a rough explanation of this 
derivation and report only the most relevant results. A detailed derivation will be published 
elsewhere in the near future. 

From Eq.( p.6| ), we easily find that the overlap at time t + 1 is given by 

m ( t+1 ) = (( / (| m(t ) + ,( t )|)^|±|) T )), (0,0) 

where ((■ • •)) represents an average over the complex Gaussian z(t) with mean and variance 
2a(t) 2 . For the noise z(t + 1), in the limit iV —>■ oo we obtain 



*(* + 1) - 4? E E tf§7(IM*)l) hjAt) 



aA^^^ J J ^ y /U \h- (t)\ 



+ z i {t)\ J U f V /I; + 



2|M*)I 



where hj tll (t) = l/aNY^k=i^v^mu,i£j£,kWk(t)- We should remark that the hj !fl (t) can be 
regarded as independent of £f . Using this fact, we can easily carry out various calculations 
for a(t + l). 
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FIG. 1. Dependence of the storage capacity a c on the mean activity level a for various threshold 
parameters H. 

First, we consider the equilibrium state of this system. In the equilibrium state we have 
Zi {t) = Zi(t + 1) = Zi (a(t) = a(t + 1) = a), m(t) = m(t + 1) = m. Using Eqs. JOJ ) and 
( P- 11| ) , we finally find that the values of m and a are determined by the equations 



m = M(m, a) 
2 aQ(m,a) 



a 



(0.12) 



2(1 -G{m,a)) 



2 ' 



with 



M(m,a) 
Q(m, a) 
+ 

G(m, a) 



dr 



*■ r cos 9 
du — exp 



m 2 + r 2 



2mr cos 9 * 



TTU 



2a 2 



a) exp 



H 2 



2a 2 



du — - exp 

H JO 7l(J 



dr 

aH 
2ixo 2 Jo 



d9exp 



roo p-K n 



exp 



m 2 + r 2 — 2mr cos 9 \ 

2o~ 2 J 
m 2 + H 2 — 2mH cos 9 N 

m 2 + r 2 — 2mr cos9\ 



(0.13) 



1 



a) 



2a 2 



H exp 



H 2 



2a 2 



2a 2 
dr exp 



2a 2 



Here, the parameter a is defined by a = P/N. If a is smaller than the maximum storage 
capacity a c , there exists a solution of Eq. (p.l2|) for which m/fl. This solution corresponds 
to a retrieval state. It disappears at a c , where the overlap m drops from the finite value m c 
to zero. In Fig. (TTJ) , the storage capacity a c obtained from numerical solutions of Eq.( p,12| ) 
is plotted as a function of the mean activity level of the memorized patterns a for various 
threshold parameters H. As shown there, the storage capacity increases as the activity level 
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decreases. Moreover, we have numerically confirmed that, like the binary model, in the limit 
a — > the storage capacity diverges as — 1/alna. 

Next, we wish to discuss the performance of the system with regard to associative mem- 
ory, that is, its associative ability to dynamically correct a noisy pattern. This measure of 
the model's performance roughly corresponds to the size of the basin of attraction, which 
is obtained from analysis of the dynamics of the recalling process. Following a previous 
work [II|, we can derived the recursion equations describing the retrieval dynamics. In or- 



der to obtain the correct results in the case of oscillator neural networks, it is essential to 
take into account the fact that the noise Zi(t) is temporally correlated. In the nth order 
approximation, the correlation of the noise Zi(t) up to Zi(t — n + 1) is correctly estimated. 
At first order (n = 1), ignoring the temporal correlation of the noise Zi(t), we obtain 

' m(t + l) = M(m(t),a(t)) 

rv 

a(t + l) 2 = -Q(m(t), a(t)) + a(t) 2 G(m(t), a{t)f (0.14) 
+a 2 aG(m(t), a(t))m(t)m(t + 1). 

This result corresponds to the Amari-Maginu theory in the case of traditional neural net- 
works 

At second order (n=2), taking account of the correlation between Zi(t) and Z{{t — 1), we 
find that 

m(t + 1) = M(m(t),a(t)) 

a(t + l) 2 = |g(m(t), a(t)) + a(t) 2 G(m(t), a{t)f 

+aG(m(t),a(t))X(t + l,t) 
+a 2 aG(m(t), a{t))G{m{t - 1), a(t - l))m{t + l)m(t - 1) (0.15) 
X(t + l,t)= Re ((aYttyY^t - 1) + (1 - a)Y (t)Y {t - 1))) 

Y k (t) = f(\6 h ,Mt) + m) ^f^fh k = 0, 1), 

\0k,im(t) + z(t)\ 

where ((• • ■)) z f t j z (t~i) represents an average over the two complex Gaussian functions z(t) 
and z(t — 1) with the correlation p(t,t — 1) = ((z(t)z(t — 1))) /2a(t)a(t — 1). In addition, 
estimating ((z(t)z(t — 1))) by using Eq.( p.ll| ), we find that p(t,t — 1) can be calculated as 

p(t,t-l)= °A-M- 1) j(i - 1)G ( m(f -!)) 

At the initial time, we choose the initial value of the overlap m(0) with the condition that 
or(O) 2 = aa/2, X(0, -1) = and A(l, 0) = a 2 m(l)m(0). 
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FIG. 2. Comparison of time evolution of overlaps with various initial conditions, 
m = 0.25,0.31,0.4 for a = 0.013, H = 0.3 and a = 0.5 . (a) A typical result of numerical 
simulations with N = 1000. (b) Theoretical curves at first order, (c) Theoretical curves at second 
order. 
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FIG. 3. The basin of attraction for the cases (H, a) = (0.3, 0.5), (0.8, 0.1) and (0.3,0.1). The 
solid lines and the broken lines represent the theoretical results at second order and at first order, 
respectively. The data points indicate simulation results with N = 1000 for 20 trials. 

In Fig.|2|, we show the time evolution of the overlap predicted by our theory and typical 
behavior found through numerical simulations. As shown in this figure, the predictions 
obtained from the second order approximation are in good agreement with numerical results. 
However, qualitative discrepancy between the first order theory and the numerical results 
can be found (See solid curves). For a < a c , only if the initial overlap is larger than a certain 
critical value m^(0), the system generally evolves toward the retrieval state m(oo) ~ 0(1). 
The size of the basin of attraction can be measured by this critical value. The nature of the 
basin of attraction is described in Fig.^|, which displays the results of theoretical analysis 
and numerical simulation as a phase diagram for various thresholds H and activity levels a. 
The upper parts and lower parts of the theoretical curves represent the equilibrium overlap 
m(oo) and the basin of attraction m^(0), respectively. It is seen that the theoretical results 
at second order are in reasonable agreement with numerical results. We thus find that, in 
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order to understand the correct behavior of the retrieval dynamics, we cannot ignore the 
effect of the temporal correlation of the noise. As a result, it is shown that, even near 
saturation, the size of the basin of attraction remains nearly as large as that in the low 
loading rate regime (a ~ 0). 

In conclusion, we have proposed a generalized model of oscillator neural networks to 
recall sparsely coded phase patterns in which some neurons are in a non-firing state and the 
other neurons encode information in the timings of spikes. In the analysis of this model, 
we have found that, as in the standard binary model, the storage capacity diverges as the 
activity level decreases. At the same time, it is found that even near saturation, the basin 
of attraction remains sufficiently wide to allow for the recall of a memorized pattern from a 
noisy trigger pattern. Hence, we can say that this system exhibits good associative memory. 
Our results suggest the potential for information processing using the timings of neuronal 
firings, though it remains to determine whether the results drawn for this model hold more 
generally. 
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